Robotic Control

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for receiving, by one or more non-real-time processors, data defining a light illumination pattern for a robotic device. Generating, by the one or more non-real-time processors and based on the data, a spline that represents the light illumination pattern, where a knot vector of the spline defines a timing profile of the light illumination pattern. Providing the spline to one or more real-time processors of the robotic system. Calculating, by the one or more real-time processors, an illumination value from the spline at each of a plurality of time steps. Controlling, by the one or more real-time processors, illumination of a lighting display of the robotic system in accordance with the illumination value of the spline at each respective time step.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.17/183,688, filed Feb. 24, 2021, which is a continuation of U.S. patentapplication Ser. No. 15/963,630, filed Apr. 26, 2018, which areincorporated herein by reference in their entirety.

BACKGROUND

Autonomous devices, such as robotic devices, are capable of producinganimated effects on light displays and coordinating motions with theanimated effects. Present development techniques to drive such functionscan be complex and may be suitable for inclusion in higher-levelapplication development for robotic devices. Furthermore, translatinghigh-level animations into hardware level control signals can beresource intensive and may require more computation resources thatexceed those of control processors for some robotic devices.

SUMMARY

In general, the disclosure relates to a process for controlling lightingdisplays and motion of a robotic device. More specifically, thedisclosure relates to a process controlling lighting displays and motionof a robotic device based on high-level program code or animation files(e.g., Adobe After Effects® files). For example, a robot can translatehigh-level programing data into a basis spline, e.g., a piecewisepolynomial parametric curve, that represents a light illuminationpattern that changes over time. The light illumination pattern mayrepresent an animated effect produced by changing light values ofindividual LEDs on the robot. The robot can calculate lighting values atparticular time steps over a duration of the lighting pattern using thebasis spline. The robot can then control individual lights on the robotin accordance with the lighting values. For example, a control processorin the robot can use the calculated lighting values to drive a lightingdisplay to produce an animation defined by the basis spline. In someimplementations, a knot vector of the spline defines a timing profile ofthe light illumination pattern. For example, the knot vector may be amathematical representation of the variation (e.g., degree of variation)in colors, brightness, and position within a display of the lightingeffects over time.

The basis spline may serve as an efficient data structure fortranslating between high-level programing data and lower-level real-timecontrol operations of the robot. For example, the basis spline providesan efficient transition between non-real time operations of the robotperformed by a first processor and real-time control operationsperformed by a second processor.

In some implementations, a robot can implement a similar process tocontrol motion. For example, the robot can translate high-levelprograming data into a basis spline that represents a pattern of motionexecuted over a period of time. The motion pattern may represent aseries of positions the robot's limb must traverse to execute themotion. The robot can calculate position values at particular time stepsover motion pattern using the basis spline. The robot can then control alimb or limbs to execute the motion in accordance with the positionvalues. For example, a control processor in the robot can use thecalculated lighting values to drive one or more motors to produce motionpattern defined by the basis spline. In some implementations, a knotvector of the spline is exposed to define a timing profile. For example,the knot vector may be a mathematical representation of variations inposition of a robotic limb within space over time and movement of thelimb as it transitions between positions. For example, the knot vectormay mathematically describe aspects of the motion such as velocity,acceleration, and jerk of the limb as it transitions between positions.

In general, innovative aspects of the subject matter described in thisspecification can be embodied in methods that include the actions ofreceiving, by one or more non-real-time processors, data defining alight illumination pattern for a robotic device; generating, by the oneor more non-real-time processors and based on the data, a spline thatrepresents the light illumination pattern, where a knot vector of thespline defines a timing profile of the light illumination pattern;providing the spline to one or more real-time processors of the roboticsystem; calculating, by the one or more real-time processors, anillumination value from the spline at each of a plurality of time steps;and controlling, by the one or more real-time processors, illuminationof a lighting display of the robotic system in accordance with theillumination value of the spline at each respective time step. Otherimplementations of this aspect include corresponding systems, apparatus,and computer programs, configured to perform the actions of the methods,encoded on computer storage devices. These and other implementations caneach optionally include one or more of the following features.

Some implementations include the steps of: receiving, by one or morenon-real-time processors, data defining a motion pattern for a roboticsystem; generating, by the one or more non-real-time processors andbased on the data, a second spline that represents the motion pattern,wherein a knot vector of the second spline defines a second timingprofile that represents timing of the motion pattern; synchronizing, bythe one or more non-real-time processors, the timing profile of thelight illumination pattern and the second timing profile; providing thesecond spline to one or more real-time processors; calculating, by theone or more real-time processors, a position value from the secondspline at each of a plurality of time steps; and controlling, by the oneor more real-time processors, a movement of the robotic device inaccordance the position value of the second spline at each respectivetime step.

In some implementations, the data defining the light illuminationpattern is an animation file.

In some implementations, the data defining the light illuminationpattern includes programing code that describes the light illuminationpattern by defining an illumination line segment for the lightingdisplay.

In some implementations, the programing code includes a start point andan end point for the line segment.

In some implementations, the programing code includes an indication of atime dependent transition of the line segment.

In some implementations, the knot vector of the spline mathematicallyrepresents the time dependent transition.

In some implementations, the calculating an illumination value from thespline at each of a plurality of time steps includes determining thattwo or more illumination line segments overlap, and calculatingillumination values for overlapping portions of the line segments basedon blending colors in the overlapping portions.

In some implementations, the plurality of time steps correspond with anoperating frequency of the one or more real-time processors.

Particular implementations of the subject matter described in thisspecification can be implemented so as to realize one or more of thefollowing advantages. Implementations may provide standardizedprograming protocols that permit developers to seamlessly controldetailed animation and motion aspects of robotic devices. For example,implementations provide a robotic system control architecture that canefficiently translate high-level programing data to specific controlsignals for driving robotic device displays and movements.Implementations may improve efficiency of complex robotic controloperations. For example, implementations distribute processingoperations between non-real-time and real-time processors to leveragetranslate high-level program data into control signals to performreal-time control of robotic operations. As another example,implementations generate splines to describe complex robotic control andpass control information between non-real-time and real-time processors.The splines represent the control information in a highly efficientstructure and reduce the complexity of data that is transferred betweenprocessors. Implementations make programing lighting motion robust tothe number of LEDs on a given robotic device. For example,implementations can process high-level programing data that is agnosticto the specific number of LEDs on a given robotic device. Thus, suchimplementations permit portability of application code across a widerange of robotic devices without adversely affecting the execution ofthe code on any particular robotic device.

The details of one or more embodiments of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram that illustrates an example of a system architecturefor controlling aspects of a robotic device.

FIG. 2 is a diagram showing different types of example light arrays fora robotic device.

FIG. 3 is a diagram that illustrates an example light animation processaccording to implementations of the present disclosure.

FIG. 4 is a diagram that illustrates another example light animationprocess according to implementations of the present disclosure.

FIG. 5 is a flow diagram that illustrates an example of a process forcontrolling aspects of a robotic device.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

FIG. 1 is a diagram that illustrates an example architecture of a system100 for controlling functions of a robotic device 110. The system 100includes a non-real-time processing system 102, a real-time processingsystem 104, and one or more output component control systems 106. Asdiscussed in more detail below, the non-real-time processing system 102receives high-level programing data, e.g., animation files or high-levelprogram code (e.g., Java code), that defines a time dependent functionof a robotic device 110. Example time dependent functions include, butare not limited to, lighting illumination patterns that vary over timeto create animated effects and patterns of motion for one or more jointsor limbs of a robotic device 110. The non-real-time processing system102 converts the high-level programing data into a spline thatrepresents the robotic device function defined by the high-levelprograming data. The spline serves as an efficient data structure fortranslating the high-level description of robotic functions intomathematical functions that can be evaluated and executed by thereal-time processing system 104. The real-time processing system 104uses the spline to segment the robotic function represented by thespline into discrete time steps. The real-time processing system 104calculates output control values for each time step that indicate adesired state of one of the robotic device's output components, e.g., alight array illumination value or a limb position value, during the timestep in order to execute the function. The real-time processing system104 controls the robotic device's output component control systems 106to drive one of more output components, e.g., individual motors orlighting elements, in accordance with the calculated control values ateach time step, thereby, executing the robotic function.

The non-real-time processing system 102 can include, but is not limitedto, a processing system of a robotic device 110 or a separate computingdevice such as a server system 112 or a user computing device 114. Usercomputing devices 114 can include, but are not limited to, desktopcomputers, laptop computers, tablet computers, or smartphones. Thereal-time processing system 104 includes one or more processors,controllers, or both that are part of a robotic device 110. Thereal-time processing system 104 provides outputs that directly controlthe real-time functions of one or more output components of the roboticdevice 110. The output component control systems 106 include interfaces,controllers, or both that drive output components of a robotic device110. For example, output component control systems can include, but arenot limited to, light driver firmware 116 for driving a light array 118,e.g., an LED array, on robotic device 110, or motor controllers 120 fordriving a system of motors 122, e.g., to control the motion of amechanical joint on a robotic device 110.

The non-real-time processing system 102 performs operations thatgenerally do not control real-time functions of a robotic device 110.For example, the non-real-time processing system 102 does not produceoutput data that is necessarily time synchronized with function of arobotic device 110. For example, the non-real-time processing system 102output data from the non-real-time processing system 102 can be storedin memory for use by the real-time processing system 104 when needed toexecute a function of the robotic device 110. In some examples,operations performed by the non-real-time processing system 102 can beconsidered as pre-processing operations. In the depicted example, thenon-real-time processing system 102 includes an application programinterface (API) 130, a light spline converter 132, a motion splineconverter 134, and a synchronizer 136. In some examples, the API 130,light spline converter 132, motion spline converter 134, andsynchronizer 136 are provided as one or more computer executableprograms, which can be executed by the non-real-time processing system102. In some implementations, some or all of the components of thenon-real-time processing system 102 can be implemented in hardware,e.g., some or all of the components can be implemented in an fieldprogrammable gate array (FPGA) or an application specific integratedcircuit (ASIC).

In some examples, the API 130 is a component of the non-real-timeprocessing system 102 that receives high-level programing data fordefining functions for robotic devices 110. For example, high-levelprograming data defining robotic functions can include animation filesor high-level programing code that define robot lighting effects ormotion. In some examples, the API 130 permits robotic program developersto control finely tuned robotic functions using high-level code,animation data files, or both. For example, the API 130 can provideprograming instructions, definitions, protocols, subroutines, or acombination thereof that permit developers to define functions, e.g.,movements and lighting animations, for robotic devices. In someexamples, the API 130 provides rules or other criteria for definingrobotic functions using animation data files, e.g., Adobe After Effects®files.

In some examples, the API 130 defines a set of robotic functionattributes that a developer can vary in order to control time dependentrobotic functions such as lighting animation patterns and motion. Insome examples, the robotic function definitions of the API 130 areagnostic to the type or characteristics of the particular hardware onindividual robotic devices that perform the desired functions. In otherwords, the API 130 definitions are independent of hardwarecharacteristics such as the type of lighting elements that on aparticular robot (e.g., LED arrays vs. LCD array), the number oflighting elements in a light array of a particular type of robot, motordrive characteristics for motors that operate robotic joints, the numberof motors a particular type of robot uses to move a joint, etc.

In some examples, the high-level programing protocols and definitionsprovided by the API 130 can serve as standardized programing protocolsfor controlling robotic joints or limbs. For example, a limb motion canbe represented by a series of positions for the limb that describe thedesired motion or part of the desired motion. In addition, transitionsbetween the positions represented by motion transitions types thatdescribe how the limb should move from one position to another. Motiontransition types can describe characteristics of the movement fromposition to position that include, but are not limited to, velocity,acceleration, and jerk. The motion transition type can indicate amathematical interpolation of motion between joint positions, which canbe represented by a spline.

For example, FIG. 2 illustrate two example light arrays of a roboticdevice. The first light array is an example of a linear light array 200and the second light array is an example of a ring light array 250. Eachof the light arrays is made up of an array of lighting elements 202. Thelighting elements 202 can be, but are not limited to, LEDs, LCD segmentsor pixels, organic LEDs (OLED), carbon nanotube display elements, orquantum dot display elements. Furthermore, various types of roboticdevices may include light arrays of different sizes with differentnumbers of lighting elements 202.

In some examples, the high-level programing protocols and definitionsprovided by the API 130 can serve as standardized programing protocolsfor driving the light arrays. For example, the API 130 can represent alight array as an indexed line in high-level programing code. Forexample, each light array 200, 250 is illustrated in FIG. 2 with a setof line indices that are normalized to range from [0] to [1] to coverthe entire array. For the linear array 200 the indices may range from[0] to [1] moving left to right across the array. For the ring array 250the indices may begin and end at the six o'clock position.

In some examples, the API 130 can permit a developer to control thelight array by defining line segments to be illuminated using theindices. For example, light array animation can be control by allowingthe developer to define a lighting pattern by controlling API providedvariables associated with one or more illumination line segments of thearray. Line segment variables can include, but are not limited to, linesegment start and end points, a time duration for illumination, anillumination color represented by an RGB value, a color transition type,an illumination intensity transition type, a sequence transition type,and an opacity value.

In some examples, a line segment for illumination is defined by itsstart and end point as represented by two indices. For example, a linesegment [0.25]-[0.75] on the linear array 200 would extend across themiddle half of the array. The same line segment (e.g., [0.25]-[0.75]) onthe ring array 250 is ambiguous and could extend along the upper orlower half of the ring. Thus, indices for a ring array may includepositive and negative values to indicate a direction around the array,e.g., counter clockwise may be positive values and clockwise may benegative values. For example, the line segment defined by [0.25]-[0.75]would represent the upper half of the ring array 250 and the linesegment defined by [−0.25]-[0.25] would represent the lower half of thering array 250.

In some examples, the start and end point can each be associated with adifferent illumination color. In such examples, the color transitiontype can indicate how the color of the line segment should transitionfrom the color of the start point to the color of the end point. Colortransition types can include, but are not limited to, lineartransitions, exponential transitions, and abrupt transitions, e.g.,step-functions. The transition type can indicate a mathematicalinterpolation of the line segment colors between the start and endpoints, which can be represented by a spline. In some examples, thecolor transition of a line segment over time can be referred to as acolor trajectory.

In some examples, temporal aspects of the line segment (e.g.,animations) can be represented by an illumination duration variable, anillumination intensity transitions type variable, or both. For example,a duration variable can indicate how long the line segment is to beilluminated. The duration variable can also be paired with a transitiontype. For example, the duration can indicate a period of time over whicha transition of the line segment should elapse. The transition type cangive the appearance that the light segment is fading in or fading out.For example, the transition variable may indicate that the line segmentis to increase in brightness (e.g., illumination intensity) linearlyover a duration of two seconds. Illumination intensity transitions typescan include, but are not limited to, linear transitions, exponentialtransitions, and abrupt transitions, e.g., step-functions. Thetransition type can indicate a mathematical interpolation of the mannerin which the line segment is to be illuminated, which can be representedby a spline.

In some examples, the illumination intensity transitions and durationcan indicate a temporal sequence for illuminating individual lightingelements in the light array. The temporal sequence can give theappearance that the line segment is moving along the array. For example,the illumination intensity transitions can indicate a timing sequencefor illuminating individual lighting elements from the start point tothe end point. The duration can indicate the total time over which thetransition should take place. Illumination sequence transition types caninclude, but are not limited to, linear sequential transitions,exponential sequential transitions, and abrupt sequential transitions,e.g., step-functions. The sequential transition type can indicate amathematical interpolation of a sequence for illuminating light elementsalong a line segment, which can be represented by a spline.

FIG. 3 illustrates an example light animation process 300 in a seriesusing a series of diagrams (302-306) of the ring array 250. For example,a line segment of the ring array 250 may be defined by a start point, anendpoint of, a sequential transition type of a linear sequence, and aduration for the transition of two seconds. Each start and end point canbe indicated by a pair of light array indices, e.g., to indicate thelength of the line segment. Further, intermediate line segment points inaddition to the startpoint and endpoint also may be defined. Thishigh-level definition of a line segment describes an illuminationpattern that illuminates each light element in the ring array 250sequentially in a counter clockwise direction over the duration of twoseconds. For instance, diagram 302 represents the ring array 250 at afirst time with the line segment animation just beginning. In diagram302, light element 310 is illuminated in red (R) at the start point ofthe line segment (e.g., index [0, 0.033]). Diagram 304 represents thering array 250 at a second time in the animation of the line segment(e.g., index [0.067, 0.167]. In diagram 304, three light elements312-316 are illuminated in red as the segment begins to traverse thering array 250. For example, the leading light element 312 may be in theprocess of turning on, the middle light element 314 may be fullyilluminated, and the trailing light element 316 may be dimming as it isturned off. In some implementations, the sequential transition type alsodescribes how each light element energizes over time (e.g., how quicklyit is faded on or off). Finally, diagram 306 represents the ring array250 at a third time in the animation of the line segment (e.g., index[0.567, 0.667]). In diagram 306, three light elements 320-324 areilluminated in red as the segment continues to traverse the ring array250 in the counterclockwise direction.

As another example, a second line segment of the ring array 250 may bedefined by a start point of [−1], an end point of [0], a sequencetransition type of a linear sequence, and a duration for the transitionof two seconds. This high-level definition of a second line segmentdescribes an illumination pattern that illuminates each light element inthe ring array 250 sequentially in a clockwise direction over theduration of two seconds.

In some implementations, line segment points greater than 1 can indicatemultiple rotations around a ring array 250. As another example, a thirdline segment of the ring array 250 may be defined by a start point of[0, 0.1], an end point of [4.9, 5], a sequence transition type of alinear sequence, and a duration for the transition of two seconds. Thishigh-level definition of a second line segment describes an illuminationpattern that illuminates each light element in the ring array 250sequentially in a counter clockwise direction for five revolutionsaround the array over the duration of two seconds.

Referring again to FIG. 1 , in some implementations, the non-real-timeprocessing system 102 preprocesses animation files. For example, someanimation files represent time as a dependent variable in functionsdefining the animation. This can create excessive processing overhead,processing errors, or both when converting a robot lighting animation orrobot motion defined in the animation file into a spline. In suchimplementations, the non-real-time processing system 102 can convert thetime related variables in the animation file from dependent variables toindependent variables. For example, the non-real-time processing system102 can read the animation file and approximate a timing profile for therobotic function. Using an Adobe After Effects® Butterfly file as anexample, the non-real-time processing system 102 can approximate aseries of cubic Bezier curves defined in the file into a single spatialspline with knot vectors that represent the timing profile of theanimation, and, by extension, the desired robotic function. For example,the non-real-time processing system 102 can use a linear least-squaresregression to minimize errors in control points generated by theapproximation. The non-real-time processing system 102 can increase thenumber of knots in the knot vector and recalculate the regression if thecontrol point error exceeds a predefined error threshold.

In some implementations, the API 130 can directly expose splinegeneration primitives. For example, the API 130 can provide programingprotocols that permit a developer to specify robotic functions byproviding a set of spline control points and knots. In suchimplementations, the spline converters 132/134 can pass the splinecontrol points and knot vector to an appropriate one of the splinetrajectory controllers 138/144 of the real-time processing system 104.In some implementations, the spline converters 132/134 can checkdeveloper defined spline control points and knots for errors prior topassing them to the real-time processing system 104.

The light spline converter 132 generates a lighting spline 133 thatrepresents a light illumination pattern defined by the high-levelprograming data. For example, the light spline converter 132 cantranslate the high-level programing data into a basis spline, e.g., athird-order basis spline, which represents changes in the lightillumination pattern over time. For example, the light spline converter132 can determine a timing profile for the light illumination patternbased on attributes of the illumination pattern including, but notlimited to, the color transition type, the illumination intensitytransitions type, and the sequence transition type. The light splineconverter 132 can represent the timing profile as a set of piecewisefunctions that mathematically describe the desired lighting animation.In other words, light spline converter 132 can generate the timingprofile based on high-level programing code that defines the lightingpattern. For example, the timing profile can be determined based on thelighting pattern transitions defined by the programing code. The timingprofile may represent, e.g., the timing of changes in color, intensity,and illumination sequences for appropriate portions of one or morelighting displays on the robotic device. In some examples, the knotvector equations are a mathematical representation of the timingprofile.

In some examples, the light spline converter 132 can translate data froman animation file into a spline representation of lighting animationdefined by the file. For example, the light spline converter 132 canextract key frames from the animation file and generate a spline basedon interpolating transitions in a lighting pattern between the keyframes.

In some implementations, a knot vector of the lighting spline 133 candefine the timing profile of the light illumination pattern to generatethe animated effect. For example, the knot vector can serve as amathematical representation of the variation in colors, brightness, andposition (within a display) of the lighting effects as a function oftime.

The motion spline converter 134 generates a motion spline 135 thatrepresents a motion pattern defined by the high-level programing data.For example, the motion spline converter 134 can translate thehigh-level programing data into a basis spline, e.g., a third-orderbasis spline, which represents a series of positions, which a roboticlimb must traverse to execute the motion. For example, the motion splineconverter 134 can determine a timing profile for the motion patternbased on the limb positions and motion transition types betweenpositions. The motion spline converter 134 can represent the timingprofile as a set of piecewise functions that mathematically describe thedesired motion of the limb. In some examples, the motion splineconverter 134 can translate data from an animation file into a motionspline 135 representation of lighting animation defined by the file. Forexample, the motion spline converter 134 can extract key frames from theanimation file. The key frames may provide a series of discretepositions for the limb to execute the animation. The motion splineconverter 134 can generate a motion spline 135 based on interpolatingmotion transitions between the key frames.

In some implementations, a knot vector of the motion spline 135 candefine the timing profile of the motion pattern. For example, the knotvector can serve as a mathematical representation of the variation inlimb position and transitions between positions as a function of time.

The synchronizer 136 synchronizes the timing of lighting and motionpatterns for robotic functions that include both lighting and motionaspects. For example, the synchronizer 136 can synchronize lightillumination and motion timing profiles between two splines to ensurethat a robotic device's 110 motion and lighting patterns are executedproperly. For instance, a particular animation sequence may require arobotic device to accelerate a limb faster than the motors of the deviceare capable of moving the limb. In such situations, the motion splineconverter 134 can adjust the acceleration of the limb as represented bya motion spline 135 to a value that is within the capabilities of therobotic device's motors. In response, the synchronizer 136 can makesimilar adjustments to the lighting spline 133 to maintainsynchronization between the desired motion and light animation.

In some examples, a desired motion function for a robotic device 110 mayrequire that a joint be moved to an initial starting position beforebeginning a motion. For example, a particular motion pattern for a limbmay require the limb be driven to a starting position. If such a motionpattern is associated with a lighting animation, the synchronizer candelay the lighting animation while repositioning a limb to ensure thatboth the motion and the lighting animation begin at the same time. Forexample, the synchronizer 136 can insert or cause the light splineconverter 132 to insert a delay period into the beginning of thelighting spline 133 that accounts for a period of time during which thelimb is repositioned.

The real-time processing system 104 controls the execution of roboticfunctions in real-time. For example, the real-time processing system 104can include one or more control processors that process lighting and/ormotions splines to control the illumination of light arrays and/or robotmotion in real-time. In the depicted example, the real-time processingsystem 104 includes a lighting spline trajectory controller 138, a lightselection controller 140, an, optional, reset controller 142, a motionspline trajectory controller 144, and a joint selection controller 146.In some examples, the lighting spline trajectory controller 138, lightselection controller 140, reset controller 142, motion spline trajectorycontroller 144, and joint selection controller 146 are provided as oneor more computer executable programs, which can be executed by thereal-time processing system 104. In some examples, the real-timeprocessing system 104 operates with a 250 Hz execution loop. In otherwords, instructions are processed and new commands or control signalsare transmitted to output hardware of the robotic device 110approximately every 4 ms. In other implementations, the real-timeprocessing system 104 operates with a 1000 Hz to 100 Hz execution loop.In some implementations, some or all of the components of the real-timeprocessing system 104 can be implemented in hardware, e.g., some or allof the components can be implemented in an FPGA or an ASIC.

The lighting spline trajectory controller 138 receives lighting splines133 from the non-real-time processing system 102 and determines discretelighting values for each light element in a light array based on one ormore lighting splines 133. For example, the lighting spline trajectorycontroller 138 can calculate an illumination value for individual lightelements of an array from a lighting spline 133. In some examples, thespline may define lighting values for every light element in an array.That is the spline may evaluate to a zero illumination value fornon-illuminated light elements. In some examples, the spline may onlydefine lighting values for light elements that are within anillumination line segment, e.g., light elements that will beilluminated. In some examples, the spline may only define lightingvalues for key light elements of an illumination line segment of a lightarray. In such examples, the lighting spline trajectory controller 138can interpolate illumination values for non-key light elements that arewithin the illuminated line segment from the illumination values of thekey light elements. Illumination values can include, but are not limitedto, a light intensity value that indicates a brightness level for alight element and a color value that indicates a color of a lightelement. The color value can be, e.g., an RGB (red-green-blue) array oran RGBW (red-green-blue-white) array.

In some implementations, the lighting spline trajectory controller 138calculates illumination values from the lighting spline 133 for lightelements of a light array 118 at discrete time steps. In some examples,the time steps can correspond to the operating frequency of thereal-time processing system 104.

In some implementations, the lighting spline trajectory controller 138can process two or more light array line segments simultaneously.Multiple illumination line segments can be defined by multiple splinesthat share a knot vector and are processed concurrently. When multipleline segments overlap on a light array 118, the lighting splinetrajectory controller 138 can blend the colors of the overlapping linesegments. For example, the lighting spline trajectory controller 138 canuse alpha blending techniques to blend overlapping portions of linesegments.

FIG. 4 provides an illustrative example of line segment blending. FIG. 4illustrates an example of a light animation and blending process 400using a series of diagrams (402-408) of the linear array 200 (from FIG.2 ). Each diagram represents the linear array as illuminated during atime step of the lighting spline trajectory controller 138. Diagram 402shows a red (R) line segment 410 traversing the linear array 200 in aleft to right direction and a green (G) line segment 412 traversing thelinear array 200 in a right to left direction. In diagram 404, the redline segment 410 and the green line segment 412 begin to intersect atlight elements 414. The lighting spline trajectory controller 138computes a yellow (Y) illumination value for light elements 414 by alphablending illumination values computed for light elements 414 based onthe lighting spline (or splines) that define the red line segment 410and the green line segment 412. In diagram 406, the red line segment 410and the green line segment 412 have passed each other as they traversethe linear array 200 in their respective directions. Thus, the greenline segment 412 is now shown on the left side of the linear array 200and the red line segment 410 is now shown on the right side of thelinear array 200. However, the two line segments still overlap at lightelement 416, the illumination values of which are alpha blended toproduce yellow. In diagram 408, the red line segment 410 and the greenline segment 412 no longer overlap; therefore, the lighting splinetrajectory controller 138 no longer blends the line segments.

Referring again to FIG. 1 , the light selection controller 140 receiveslight element illumination values from the lighting spline trajectorycontroller 138 and provides control signals to the firmware 116 to drivethe light arrays 118 on a robotic device 110. For example, the lightselection controller 140 receives illumination values from the lightingspline trajectory controller 138 during each time step. The lightselection controller 140 parses the illumination values and can sendcontrol signals to firmware 116 to drive appropriate light elements ofthe light arrays 118.

Referring again to FIG. 1 , the reset controller 142 provides non-splinebased control of the light arrays 118. For example, the reset controller142 can provide forced controls or manual controls for the light array118. The reset controller 142 can be used to reset one or more lightarrays 118 by, e.g., turning all light elements off or to a predefinedillumination value (e.g., white).

The motion spline trajectory controller 144 receives motion splines 135from the non-real-time processing system 102 and determines discreteposition values for one or more limbs or joints of the robotic device110. For example, the motion spline trajectory controller 144 cancalculate position values for individual limbs of the robotic device 110from the motion spline 135. In some implementations, the motion splinetrajectory controller 144 calculates position values from the motionspline 135 for the robotic device 110 at discrete time steps. In someexamples, the time steps can correspond to the operating frequency ofthe real-time processing system 104.

The joint selection controller 146 receives position values from themotion spline trajectory controller 144 and provides control signals tothe motor controllers 120 to drive the motors 122 to move appropriatelimbs of the robotic device 110. For example, the joint selectioncontroller 146 receives position values from the motion splinetrajectory controller 144 during each time step. The joint selectioncontroller 146 selects appropriate joints and motors to operate in orderto move associated limbs from one position to another based on theposition values. The joint selection controller 146 provides controlsignals to the motor controller 120 to drive the motors 122 that operatethe robotic device's 110 limbs.

FIG. 5 is a flow diagram that illustrate a process 500 for controlling arobotic device or a robotic system. The process 500 can be performed byone or more processing systems. For example, a robotic system caninclude a non-real-time processing system and a real-time processingsystem. Both processing systems may reside on a single robotic device orthe robotic system may include a robotic device with a real-timeprocessing system in communication with a non-real-time processingsystem that resides on another computing device (e.g., server system oruser commuting device). As discussed above, some of the operations maybe performed by a non-real-time processing system and some operationsmay be performed by a real-time processing system. Each set of steps502-508 and steps 510-516 represent separate sub-process that can beperformed separately or together. Step 520 is performed to synchronizethe sub-processes, e.g., when a light animation and motion need to besynchronized.

The system receives data defining a light illumination pattern for arobotic device (502). For example, the light illumination pattern is atime dependent lighting pattern to produce an animated effect on alighting display of the robotic device. The lighting display caninclude, e.g., one or more light arrays. The data can include high-levelprograming data that defines the light illumination pattern. Forexample, the data can be high-level programing code or an animationfile. The data defines the characteristics of the lighting patternincluding, but not limited to, illumination line segments, colors,opacity, illumination intensity, e.g., brightness, and lighting patterntransitions. Lighting pattern transitions can be time dependent orspatial. Lighting pattern transitions can include, but are not limitedto, color transitions, illumination intensity transitions, and sequencetransitions. Transitions can be defined by programing data indicating adesired transition type between states of the pattern and can include,but are not limited to, linear transitions, exponential transitions,geometric transitions, and abrupt transitions, e.g., step-functions.

The system generates a lighting spline that represents the lightillumination pattern (504). For example, the system can generate aspline that mathematically represents the light illumination pattern asdefined by the data received in step 502. For example, the system cantranslate the high-level programing data into a spline thatmathematically defines the light illumination pattern over time. Thesystem can determine a timing profile of the illumination pattern basedon the high-level programing data and generate a spline with knotfunctions that represent the timing profile. For example, the system cangenerate the timing profile based on high-level programing code thatdefines the lighting pattern. For example, the timing profile can bedetermined based on the lighting pattern transitions defined by theprograming code. The timing profile may represent the timing of changesin color, intensity, and illumination sequences for appropriate portionsof one or more lighting displays on the robotic device. In someexamples, the knot vector equations are a mathematical representation ofthe timing profile. As another example, the system extract key framesfrom the animation file and generate a spline based on interpolatingtransitions in the lighting pattern between the key frames.

The system calculates illumination values from the lighting spline(506). For example, the system calculates illumination values forportions of the light array(s) at discrete time steps based on thelighting spline. The lighting values may include a color value and anintensity value. For example, the color value can be an RGB or an RGBWarray. In some examples, the time steps correspond to an operationfrequency of one or more real-time processors. For example, the systemcan calculate illumination values at intervals that correspond to anoperational period of the one or more real-time processors (e.g., theinverse of the operation frequency).

The system controls a lighting display on the robotic device to producethe lighting pattern (508). For example, the system can generate controlsignals for individual light elements of the light array(s) based on theillumination values. The system can provide the control signals tofirmware that drives the light array(s) to produce a portion ofillumination pattern during each time step.

The system receives data defining a motion pattern for the roboticdevice (510). For example, the motion pattern is a time dependentpattern to control movement of a limb of the robotic device, a joint ofthe robotic device, movement of the robotic device itself, or acombination thereof. The data can include high-level programing datathat defines the light illumination pattern. For example, the data canbe high-level programing code or an animation file. The data defines thecharacteristics of the motion pattern including, but not limited to,positions of a limb, joint, or the robotic device and transitionsbetween positions. For example, motion transitions can describecharacteristics of the movement from position to position that include,but are not limited to, velocity, acceleration, and jerk. The motiontransition can indicate a mathematical interpolation of motion betweenjoint positions, which can be represented by a spline. Motiontransitions can be defined by programing data indicating a desiredtransition type between positions of the motion pattern and can include,but are not limited to, linear transitions, exponential transitions,geometric transitions, and abrupt transitions, e.g., step-functions.

The system generates a motion spline that represents the motion pattern(512). For example, the system can generate a spline that mathematicallyrepresents the motion pattern as defined by the data received in step510. For example, the system can translate the high-level programingdata into a spline that mathematically defines the motion pattern overtime. The system can determine a timing profile of the motion patternbased on the high-level programing data and generate a spline with knotfunctions that represent the timing profile. For example, the system cangenerate the timing profile based on high-level programing code thatdefines the motion pattern. For example, the timing profile can bedetermined based on the positions and the motion pattern transitionsdefined by the programing code. The timing profile may represent thetiming of changes in position. In some examples, the knot vectorequations are a mathematical representation of the timing profile. Asanother example, the system extracts key frames from the animation fileand generate a spline based on interpolating transitions in the motionpattern between the key frames.

The system calculates position values from the motion spline (514). Forexample, the system calculates position values for the robotic device atdiscrete time steps based on the motion spline. In some examples, thetime steps correspond to an operation frequency of one or more real-timeprocessors. For example, the system can calculate position values atintervals that correspond to an operational period of the one or morereal-time processors (e.g., the inverse of the operation frequency).

The system controls a movement of the robotic device (516). For example,the system can generate control signals for individual motors oractuators of the robotic device to produce the desired motion pattern.The system can provide the control signals to motor controllers and/orfirmware that drives the limbs or joints of the robotic device.

In some examples, the system synchronizes the lighting spline and themotion spline to synchronize the lighting animation with the motion ofthe robotic device (520). For example, the system can synchronize thetiming of lighting and motion patterns for robotic functions thatinclude both lighting and motion aspects. For example, the system cansynchronize light illumination and motion timing profiles between twosplines to ensure that a robotic device's motion and lighting patternsare executed in unison. For example, the system can adjust the timing ofone or both of the splines to ensure that the lighting and motionpatterns are performed by the robotic device in unison.

Embodiments of the invention and all of the functional operationsdescribed in this specification may be implemented in digital electroniccircuitry, or in computer software, firmware, or hardware, including thestructures disclosed in this specification and their structuralequivalents, or in combinations of one or more of them. Embodiments ofthe invention may be implemented as one or more computer programproducts, i.e., one or more modules of computer program instructionsencoded on a computer-readable medium for execution by, or to controlthe operation of, data processing apparatus. The computer readablemedium may be a non-transitory computer readable storage medium, amachine-readable storage device, a machine-readable storage substrate, amemory device, a composition of matter affecting a machine-readablepropagated signal, or a combination of one or more of them. The term“data processing apparatus” encompasses all apparatus, devices, andmachines for processing data, including by way of example a programmableprocessor, a computer, or multiple processors or computers. Theapparatus may include, in addition to hardware, code that creates anexecution environment for the computer program in question, e.g., codethat constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them. A propagated signal is an artificially generated signal, e.g.,a machine-generated electrical, optical, or electromagnetic signal thatis generated to encode information for transmission to suitable receiverapparatus.

A computer program (also known as a program, software, softwareapplication, script, or code) may be written in any form of programminglanguage, including compiled or interpreted languages, and it may bedeployed in any form, including as a stand-alone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program may be stored in a portion of a filethat holds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub programs, or portions of code). A computer programmay be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this specification may beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows may also be performedby, and apparatus may also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto optical disks, or optical disks. However, a computerneed not have such devices. Moreover, a computer may be embedded inanother device, e.g., a tablet computer, a mobile telephone, a personaldigital assistant (PDA), a mobile audio player, a Global PositioningSystem (GPS) receiver, to name just a few. Computer readable mediasuitable for storing computer program instructions and data include allforms of non-volatile memory, media, and memory devices, including byway of example semiconductor memory devices, e.g., EPROM, EEPROM, andflash memory devices; magnetic disks, e.g., internal hard disks orremovable disks; magneto optical disks; and CD ROM and DVD-ROM disks.The processor and the memory may be supplemented by, or incorporated in,special purpose logic circuitry.

To provide for interaction with a user, embodiments of the invention maybe implemented on a computer having a display device, e.g., a CRT(cathode ray tube) or LCD (liquid crystal display) monitor, fordisplaying information to the user and a keyboard and a pointing device,e.g., a mouse or a trackball, by which the user may provide input to thecomputer. Other kinds of devices may be used to provide for interactionwith a user as well; for example, feedback provided to the user may beany form of sensory feedback, e.g., visual feedback, auditory feedback,or tactile feedback; and input from the user may be received in anyform, including acoustic, speech, or tactile input.

Embodiments of the invention may be implemented in a computing systemthat includes a back end component, e.g., as a data server, or thatincludes a middleware component, e.g., an application server, or thatincludes a front end component, e.g., a client computer having agraphical user interface or a Web browser through which a user mayinteract with an implementation of the invention, or any combination ofone or more such back end, middleware, or front end components. Thecomponents of the system may be interconnected by any form or medium ofdigital data communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), e.g., the Internet.

The computing system may include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of the invention or of what may beclaimed, but rather as descriptions of features specific to particularembodiments of the invention. Certain features that are described inthis specification in the context of separate embodiments may also beimplemented in combination in a single embodiment. Conversely, variousfeatures that are described in the context of a single embodiment mayalso be implemented in multiple embodiments separately or in anysuitable subcombination. Moreover, although features may be describedabove as acting in certain combinations and even initially claimed assuch, one or more features from a claimed combination may in some casesbe excised from the combination, and the claimed combination may bedirected to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems maygenerally be integrated together in a single software product orpackaged into multiple software products.

In each instance where an HTML file is mentioned, other file types orformats may be substituted. For instance, an HTML file may be replacedby an XML, JSON, plain text, or other types of files. Moreover, where atable or hash table is mentioned, other data structures (such asspreadsheets, relational databases, or structured files) may be used.

Thus, particular embodiments of the invention have been described. Otherembodiments are within the scope of the following claims. For example,the actions recited in the claims may be performed in a different orderand still achieve desirable results.

What is claimed is:
 1. A method of controlling a robotic system, themethod comprising: receiving, by one or more processors of the roboticsystem, a spline that represents a motion pattern, wherein a knot vectorof the spline defines a timing profile that represents timing of themotion pattern; determining, by the one or more processors of therobotic system and based on the knot vector, a position value of thespline for each of a plurality of time steps; and controlling, by theone or more processors of the robotic system, a movement of the roboticsystem in accordance with the position value of the spline determinedfor each of the plurality of time steps.
 2. The method of claim 1,wherein the spline is defined by an animation file.
 3. The method ofclaim 1, wherein the spline is defined by programing code describing themotion pattern by defining a series of positions for a component of therobotic system.
 4. The method of claim 3, wherein the programing codeincludes transition types indicating characteristics of how the roboticsystem should move the component between respective positions.
 5. Themethod of claim 3, wherein the programing code includes an indication ofa time dependent transition between two respective position values ofthe spline.
 6. The method of claim 1, wherein the knot vector of thespline mathematically represents time dependent transitions betweenrespective position values of the spline.
 7. The method of claim 1,wherein the knot vector of the spline mathematically representscharacteristics of the movement of the robotic system between respectiveposition values of the spline.
 8. The method of claim 7, wherein thecharacteristics of the movement include one or more of a velocity, anacceleration, or a jerk of the robotic system.
 9. The method of claim 1,further comprising receiving a second spline that represents a lightingpattern synchronized with the motion pattern, and wherein the methodfurther comprising controlling a lighting effect of the robotic systembased on the second spline.
 10. A robotic system comprising: a real-timeprocessing system; and one or more storage devices storing instructionsthat are operable, when executed by the real-time processing system, tocause real-time processing system to perform real-time operationscomprising: receiving a spline that represents a motion pattern, whereina knot vector of the spline defines a timing profile that representstiming of the motion pattern; determining, based on the knot vector, aposition value of the spline for each of a plurality of time steps; andcontrolling a movement of the robotic system in accordance with theposition value of the spline determined for each of the plurality oftime steps.
 11. The robotic system of claim 10, wherein the spline isdefined by an animation file.
 12. The robotic system of claim 10,wherein the spline is defined by programing code describing the motionpattern by defining a series of positions for a component of the roboticsystem.
 13. The robotic system of claim 12, wherein the programing codeincludes transition types indicating characteristics of how the roboticsystem should move the component between respective positions.
 14. Therobotic system of claim 12, wherein the programing code includes anindication of a time dependent transition between two respectiveposition values of the spline.
 15. The robotic system of claim 10,wherein the knot vector of the spline mathematically represents timedependent transitions between respective position values of the spline.16. The robotic system of claim 10, wherein the knot vector of thespline mathematically represents characteristics of the movement of therobotic system between respective position values of the spline.
 17. Therobotic system of claim 16, wherein the characteristics of the movementinclude one or more of a velocity, an acceleration, or a jerk of therobotic system.
 18. The robotic system of claim 10, the operationsfurther comprising receiving a second spline that represents a lightingpattern synchronized with the motion pattern, and wherein the operationsfurther comprising controlling a lighting effect of the robotic systembased on the second spline.
 19. One or more storage devices storinginstructions that are operable, when executed by a real-time processingsystem, to cause real-time processing system to perform real-timeoperations comprising: receiving a spline that represents a motionpattern, wherein a knot vector of the spline defines a timing profilethat represents timing of the motion pattern; determining, based on theknot vector, a position value of the spline for each of a plurality oftime steps; and controlling a movement of a robotic system in accordancewith the position value of the spline determined for each of theplurality of time steps.
 20. The one or more storage devices of claim19, the operations further comprising receiving a second spline thatrepresents a lighting pattern synchronized with the motion pattern, andwherein the operations further comprising controlling a lighting effectof the robotic system based on the second spline.